DOCUMENT RESUME 



ED 460 428 



EA 029 699 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Waters, Gisele A.; Ares, Nancy 

Voices of Power, Equity, and Justice in the Evaluation of 
Education Reform. 

1998-11-00 

35p.; Paper presented at the Annual Meeting of the Mid-South 
Educational Research Association (27th, New Orleans, LA, 
November 4-6, 1998). 

Reports - Research (143) -- Speeches/Meeting Papers (150) 
MF01/PC02 Plus Postage. 

♦Educational Assessment; Educational Change; Elementary 
Secondary Education; Evaluation Criteria; *Program 
Evaluation; Program Validation; School Effectiveness; 

♦Values 

Stakeholders 



ABSTRACT 

This paper reviews evaluations of social programs and major 
approaches to program evaluation. Its purpose is to examine the roles that 
philosophy, ideology, key audiences,, preferred methods, and typical 
evaluation questions can play in the practices of contemporary evaluators. 

The report describes the Learning Connections Projects, a program that can 
provide a context for the philosophical framework of the pragmatist and 
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reform by first facing their ideologies and then reflecting on the 
situational and generalized ethics that apply to any given research act. 
Lastly, consequences and benefits of using a critical normative framework are 
explored as a way to negotiate and to communicate the social and political 
meanings of evaluation questions and ideological and value orientations with 
high-level policy makers, program beneficiaries, potential clients, and 
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Voices of Power, Equity, and Justice in the Evaluation of Education Reform 

Abstract 

This paper reviews evaluation of social programs and major approaches to program evaluation in 
order to examine the roles that philosophy, ideology, key audiences, preferred methods, and 
typical evaluation questions can play in the practices of contemporary evaluators. The Learning 
Connections Project is described as a context for the philosophical framework of the pragmatist 
and developmental approaches to evaluation. Next, a broad discussion explores how evaluation 
approaches can serve particulars set of social and political values. Following this discussion, the 
school reform movement is summarized to show how it has done little to provide an accurate 
analysis of the productiori of inequality in the public schools or the larger social order. This paper 
then suggests how evaluators can earn a seat at the decision making table of public school reform 
by first facing their ideologies and reflecting on the situational and generalized ethics that apply 
to any given research act. Lastly, consequences and benefits of using a critical normative 
framework is explored as a way to explicitly negotiate and communicate the social and political 
meanings of evaluation questions and ideological and value orientations with high level policy 
makers, program beneficiaries, potential clients, and reformers. 
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Voices of Power, Equity, and Justice in the Evaluation of Education Reform 

Introduction 

Denzin and Lincoln (1998) state that evaluations are conducted on social programs, most 
importantly, on social programs in the public domain. Social programs such as public education 
are manifest responses to priority individual and community needs (Denzin & Lincoln, 1998). 
The work of social program evaluators is framed by the concerns and interests of selected 
members of the setting being evaluated. Denzin and Lincoln (1998) also argue that evaluation 
questions about the significance of program goals or about the quality and effectiveness of 
program strategies do not reflect objective inquiry or theoretical predictions, but rather a 
politicized process of priority setting. In all evaluation contexts, there are multiple, often 
competing, potential audiences, groups and individuals who have vested interests in the program 
being evaluated, called stakeholders in evaluation jargon. These stakeholders range from policy 
makers and funders, to program administrators and staff, to intended beneficiaries such as 
teachers, students, parents, and the citizenry at large. Therefore, unlike most other social 
scientists who assume an audience of peers or scholars, evaluators must negotiate whose 
questions will be addressed and whose interests will be served by their work. 

Jennifer Greene (1998) argues that neither the diverse criteria for program effectiveness 
nor the different stakeholders’ widely divergent evaluation questions can be equally well 
addressed by the same evaluation methods or approaches. In this respect, it is the fundamentally 
political nature of program evaluation contexts, intertwined \vith the predispositions and beliefs 
of the evaluator, that shape the contours of evaluation methodologies and guide the selection of a 
specific evaluation approach for a given context. Different evaluation methodologies are 
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expressly oriented around the information needs of the different audiences in social programs, 
from the macro-level program and cost effectiveness questions of policy makers and funders, to 
the micro-level questions of meaning for individual participants (Greene, 1998). These varied 
audience orientations further represent, explicitly or implicitly, the promotion of different socio- 
political values and stances. Evaluation approaches hence constitute coordinated frameworks of 
philosophical assumptions (about the world, human nature, knowledge, ethics), integrated with 
ideological views about the role and purpose of social inquiry in policy and program decision 
making. Value judgments regarding the desired ends of programs accompany those philosophical 
assumptions and ideological views, which then shape evaluation methodologies (Denzin & 
Lincoln, 1998). 

A Review of Program Evaluation Approaches 

The following section describes two categorizations of major evaluation approaches, one 
proposed by Greene (1998) and the other by Chelimsky & Shadish (1997). Greene (1998) creates 
four categories: postpositivist, pragmatist, interpretivist, and critical normative science. She 
bases her formulation on the philosophical frameworks, ideologies, key audiences, preferred 
methods, and typical evaluation questions characteristic of each approach. Chelimsky and 
Shadish (1997) classify evaluation approaches into three categories — knowledge, 
accountability, development — based upon the purposes and questions that guide them 

In Greene’s (1998) classification, the postpositivist philosophical framework represents 
the historically dominant tradition in program evaluation. This approach is ideologically oriented 
around the macro-level policy issues of program effectiveness and cost efficiency. These 
evaluations focus on technical outcomes and on the social values of efficiency, accountability, 
and theoretical causal knowledge. The key audiences include high level policy makers and 
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funders. The preferred methods of a postpositivist evaluator are quantitative experiments and 
quasi-experiments, including systems analysis, causal modeling, and cost-benefit analyses. Some 
typical questions answered by this approach would be, “Are desired outcomes attained and 
attributable to the program?”, or “Is this program the most efficient alternative?” (Greene, 1998, 
p. 376). Postpositivist evaluators retain a strong position among evaluation practice and, perhaps 
most notably, evaluation audiences, despite their well documented failure to meet the demands of 
social programs (Patton, 1997). 

The second approach Greene (1998) describes is the pragmatist philosophical framework 
and represents a “response to the failure of experimental science to provide timely and useful 
information for program decision making” (p. 377). This approach is ideologically oriented 
around management and quality control, and is focused on the values of practicality and utility. 
The key audiences are mid-level program managers, administrators, and other decision makers. 
The preferred methods of a pragmatist evaluator are a mix of quantitative and qualitative data, 
including structured and unstructured surveys, questionnaires, interviews, and observations. 
Typical questions answered by this approach would be, “Which parts of the program work well 
and which need improvement?” and “How effective is the program with respect to the 
organization’s goals? With respect to the beneficiaries’ needs?”(Greene, 1998, p. 376). 

Evaluators of this approach select their methods to match the practical problem at hand, rather 
than as explicitly dictated by some abstract set of philosophical tenets (Green, 1998). 

Qualitative approaches in evaluation can mostly be found in the third cluster of evaluation 
frameworks (Green, 1998). The interpretivist approach shares a common grounding in a basically 
interpretive philosophy of science, one that supports the notion that “in the social sciences there 
is only interpretation. Nothing speaks for itself Confronted with a mountain of impressions. 
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documents, behaviors, and actions, the interpretivist scientist acknowledges the challenge of 
making sense of what has been learned” (Denzin & Lincoln, 1998, p.313). The core of this 
sensibility is doubt that any discourse, research, or inquiry has a privileged place, any method or 
theory a universal and general claim, to authoritative knowledge (Richardson, 1991). The 
ideological and value orientations of this approach characteristically promote pluralism, 
understanding, diversity, and solidarity in evaluation contexts. A qualitative case study is the 
preferred method with an accompanying reliance on interviews, observations, and document 
review. The key audiences that are best served by this type of an approach are social program 
directors, staff, and beneficiaries, such as principals, parents, teachers. A typical evaluation 
question under this framework could be, “How is the program experienced by various 
stakeholders?” (Greene, 1998, p. 376). The focus on stakeholders’ understanding of the impacts 
of their efforts provided impetus for the interpretivist approach, as opposed to postpositivist 
evaluations that respond to a different audience. 

Finally, the fourth approach— critical normative science — represents a fairly recent 
development in evaluation. The feminist, neo-Marxist, critical, and other theorists in this 
framework “promote openly ideological forms of inquiry that seek to illuminate historical, 
structural, and value bases of social phenomenon and, in doing so, to catalyze political and social 
change toward greater justice, equity, power, and democracy” (Greene, 1998, p. 313). This 
approach to evaluation of educational programs seeks to provide information about the systemic 
and situational factors that limit opportunity and create or maintain inequity. The goal is to foster 
action by stakeholders that is focused on increasing educational opportunity and equity for 
disadvantaged students. Other key audiences could be the communities in which the students live 
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or other marginalized groups. The explicit ideological and value orientations that this approach 
emphasizes are emancipation, empowerment, and social change. 

According to Denzin (1998), “the age of putative value-free social science appears to be 
over” (p. 3 1 5). He also believes that the criteria for evaluation will move toward “articulat[ing] 
emancipatory, participative perspectives on the human condition and its betterment” (p.3 1 5), 
especially as further social crisis envelope our advanced pluralistic capitalistic society with 
increasing disparate value systems. This kind of approach is similar to Guba and Lincoln’s 
(1989) fourth generation evaluation which promotes an activist philosophy. The preferred 
methods of a critical normative evaluator are to include stakeholders in all aspects of the 
evaluation, including design, data collection and analysis, and reporting (Guba & Lincoln, 1989). 
Those designs typically involve structured and unstructured quantitative and qualitative methods, 
historical analyses, and social criticism. Questions addressed by this type of an approach could 
be, “In what ways are the premises, goals, or activities of the program serving to maintain power 
and resource inequities in society?, and “In what ways are the goals and objectives of the 
program serving to promote personal, social, and political gains?”(Greene, 1998, p. 376). For a 
variety of reasons, including the youth of the approach and the difficulties of gaining access to 
programs which desire such critical scrutiny, most work in this approach is rhetorical, although 
some normative evaluation has been done (Fine, 1988; Greene, 1991; McTaggart, 1990; Sirotnik 
& Oakes, 1990). 

As stated earlier, Chelimsky and Shadish (1997) classify evaluation approaches into three 
categories; their formulation represents a broader, more general mainstream perspective found in 
current evaluation literature. These three categories are those of accountability, knowledge, and 
development. By looking at these three perspectives and their positions along a spectrum, one 
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can begin to understand how different approaches and frameworks provide diverse backgrounds 
for evaluation methods. 

Chelimsky and Shadish (1997) offer a lengthy list of possible purposes for evaluation. 
These purposes include the following: (a) to measure and account for the results of public policy 
and programs, (b) to determine the efficiency of programs and their component processes, (c) to 
gain explanatory insight into social and other public problems, (d) to understand how 
organizations learn, (e) to strengthen institutions and improve managerial performance, (f) to 
increase agency responsiveness to the public, (g) to reform governments through the free flow of 
evaluative information, and (h) to expand results or efficiency measurement from that of local or 
national interventions to that of global interventions such as reducing poverty and hunger or 
reversing patterns of environmental degradation (p. 11). All of these purposes are, of course, 
worthwhile and legitimate reasons for conducting evaluations, but they differ with regard to the 
questions they address and the kinds of methods needed to answer the questions. 

Chelimsky and Shadish (1997) propose that these different purposes, along with the 
questions they seek to answer, seem to fall naturally into three general perspectives: 

*evaluation for accountability (e.g., the measurement of results or efficiency); 
*evaluation for knowledge (e.g., the acquisition of a more profound understanding 
in some specific area or field); and 

*evaluation for development (e.g., the provision of evaluative help in order to 
strengthen institutions). 

The methods of these three perspective are not mutually exclusive, and each of these three 
approaches may be needed at particular times or policy points and not at others (e.g., evaluation 
for knowledge may need to precede accountability). Chelimsky and Shadish (1997) write that 
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their formulation appears to have considerable explanatory power with regard to the current 
tension in the evaluation field. 

The Accountability Perspective 

From the standpoints of auditors, government sponsors of evaluation studies, donors to 
international organizations, and many others, evaluation is done to establish accoimtability. This 
involves the provision of information to decision makers, whether they are in the public or 
private sector. Specific cause and effect questions about the results in an accoimtability 
perspective might be: What happened to poverty levels among the very poor as a result of 
development assistance provided? Did an educational intervention or program produce more 
“effective” learning for all learners? Has teacher training increased student achievement? 

Sometimes, questions about the results from an accountability perspective may involve 
merely documentation of whether or not anything has changed after something new has been 
tried Chelimsky & Shadish, 1997). Normally, however, the ability to say that something is in fact 
a result of an intervention or program hinges on the ability to establish that it did not come about 
because of something else. Many methods are used to answer these kinds of accountability 
questions including: randomized designs, quasi-experimental designs, mixed multi-level designs, 
mixed qualitative/quantitative designs, case studies, process studies, and research synthesis 
designs. 

The Knowledge Perspective 

In the view of many researchers working independently in universities and other 
evaluators in scientific institutions, evaluation is done to generate understanding and explanation. 
Chelimsky and Shadish (1997) do not provide examples of the specific questions posed in this 
approach, given that it is the evaluator who decides what will be asked and answered, and the 
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topic generally follows from the researcher’s prior work. They explain that the evaluations 
associated with individual academic researchers, or those of research teams, will be more likely 
to continue in depth cumulative inquiry into particular areas or sectors of research rather than 
being concerned with applying systematic research methods to a variety of sectors, as with 
accountability and developmental evaluations. 

The larger purpose of the knowledge perspective is to increase understanding about the 
factors underlying public problems, about the match between these factors and the policy or 
program solutions proposed, and about the theory and logic (or lack thereof) that lie behind an 
implemented intervention. “These evaluations may employ any of the methods discussed above, 
separately or in conjunction with each other, but the purpose of knowledge gain leads logically to 
the use of the strongest designs as well as the greatest clarity possible in explication and 
documentation of methods to facilitate replication or later use in research synthesis and policy 
formulation” (Chelimsky &, Shadish, 1997, p.l4). 

The Developmental Perspective 

For government reformers, public managers, and others, evaluation is done to improve 
institutional performance. It serves as a flexible tool that works: (a) to improve the design of 
projects, (b) to measure and recommend changes in organization activities, c) to develop the 
indicators and performance targets needed to improve institutional effectiveness and 
responsiveness, (d) to monitor, in an ongoing way, how projects are being implemented across a 
number of different sites, and/or (e) to find out how beneficiaries feel about an agency and its 
programs (Chelimsky & Shadish, 1997, p. 17). To some accountability or knowledge perspective 
evaluators, developmental evaluators may seem more like evaluation consultants than evaluators, 
but those who do developmental work are convinced that building evaluation capability is as 
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important an evaluation function as evaluation itself and that, in some cases, evaluation cannot be 
done without capacity building. 

Specific questions asked of evaluators in a developmental perspective might include the 
following: What is the best research evidence with respect to formulating a new program or 
modifying an old one? How can projects be structured so that they produce evidence of the value 
of the intervention being tested? What is the most appropriate agenda for the agency? (Chelimsky 
& Shadish, 1997, p. 18). Both process and outcome designs may be used in a developmental 
perspective, depending on the evaluation question posed. In addition to the methods mentioned 
earlier, the formative methods used in the developmental perspective include the following: 
monitoring, empowerment evaluation, cluster evaluation, performance measurement, and 
research synthesis of both qualitative and quantitative methods (Chelimsky & Shadish, 1997). A 
developmental evaluator becomes part of the design team, helping to shape both processes and 
outcomes in an evolving, rapidly changing environment of constant interaction, feedback, and 
change. Using mixed methods and multiple criteria is characteristic of evaluations in this 
perspective. 

Classifying the Learning Connections Project Evaluation 

The Social Program 

Learning Connections is a three year, multi-level, integrated professional development 
and school improvement program being piloted in 5 middle schools in New Orleans, Louisiana. 
“It is designed to help teachers, administrators, students, parents and stakeholders as they build 
authentic learning communities where all learners are challenged to grow and develop”(CDL, 
1998). Learning Connections is directed by the Center for Development and Learning (CDL) in 
Covington, Louisiana. Funding for the Learning Connections Project is provided by the non- 
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profit foundation Baptist Community Ministries (BCM) in New Orleans. Funding for the 

Evaluation study, an external study being completed by the Auburn University Evaluation team, 

is also provided by the Baptist Community Ministries. The Center for Development and Learning 

describes Learning Connections Project as follows: 

The central goal of Learning Connections is to help each school community evolve into a happy, 
productive, and successful place where all students can learn and develop every day. Learning Connections 
is an intervention and change model worthy of replication in public and nonpublic schools serving all 
socioeconomic and ethnic groups. Learning Connections is based on the premise that all students want to 
learn, can learn, and deserve to learn by being taught in ways that improve their chances for success. 
Learning Connections will help school communities examine, model, and incorporate relevant research in 
the following areas: (1) building positive school and classroom climates, (2) interactive elements of 
learning as a process, (3 ) authentic, learner-centered teaching practices, and (4) building school 
communities that foster learning” (CDL, 1998, p.l). 

The Founder/Executive Director and the Clinical Director for the Center for Development and 
Learning are responsible for making high and mid-level decisions for the Learning Connections 
Project and they work closely with the Auburn University Evaluation team on many aspects of 
the evaluation, including formulating the objectives to be addressed, collaborating in data 
collection, and soliciting formative feedback from the evaluators, and negotiating the format of 
reports. 

The Auburn University Evaluation Study 



The plan of research and evaluation for Learning Connections involves a multi-level, 
multi-method approach designed to address each of the outcomes expected from the Projects. 

The outcome evaluation comprises a mixed qualitative/quantitative design. Various forms of data 
are collected for this study: school, teacher, and student survey data, individual and focus group 
inter\'iew data, classroom obser/ation data, standardized test scores, and school grades. Data 



collection and organization is a group effort between the Auburn University research team and 
the Clinical Director with other CDL staff in Louisiana. Descriptive and multivariate statistical 
procedures are completed on quantitative data to determine how the various project components 
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influence school improvement. Content analysis is conducted using the classroom observations 
and interview transcripts to provide qualitative descriptions of the impacts and effectiveness of 
the Project. In combination with quantitative data, the observations and interview data can yield 
rich, deep data about the effects of the reform effort. Particular outcomes that we investigated in 
the first year of the Project are participants’ perceptions of changes in instructional practices, in 
staff development activities, in students’ engagement in learning, and in parents’ involvement in 
schools. 

Learning Connections as a Pragmatist. Developmental Evaluation Approach 

For the most part, the outcome evaluation study of Learning Connections characterizes 
the pragmatist and developmental approaches to program evaluation, mainly because of the kinds 
of questions that are asked about the program’s effectiveness and because of its influences on the 
administrators and mid-level decision makers. Some components and project outcomes reflect 
the typical concerns of a more postpositivist or accountability approach (e.g., attributing 
outcomes to Project activities) and their key audiences such as high level policy decision makers 
and funders (e.g.. Executive Director, Clinical Director, and the funders). On the other hand, 
some aspects of the data collection and analysis reflect the key audiences (teachers, principals, 
parents & students) and preferred methods of the interpretivist approach (semi-structured 
interviews, theme analysis of classroom observations). 

In addition, in order to decide what action to take in the face of the effects of the Learning 
Connections Project, the Clinical Director and other learning specialists are likely to need a 
variety of qualitative and quantitative information related to the experiences and contexts of the 
school and classroom environment, as was delineated in the evaluation study. This decision 
making process of using a variety of information reflects the preferred methods of a pragmatist 
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approach. Also the ideological and value orientations of the evaluation study promote 
understanding and solidarity in the evaluation context. Some of the Auburn University evaluation 
questions asked are, “How are different parts of the program working and which ones need 
improvement?” and “How effective is the program with respect to the organization’s goals? 

These questions are similar to the examples of questions given under the framework of the 
pragmatist approach. In other words, the evaluation study designers pragmatically selected their 
methods to match the practical problem at hand of determining whether outcomes and influences 
were being attained. 

Finally, the relationship the evaluation team has developed with the Learning 
Cormections staff over the first year of the project places our approach in the developmental 
category. Our involvement in the early phases of the Project included participating in the 
formulation of objectives, based on the stated goals of Learning Cormections. We have given 
ongoing feedback to Project staff throughout the year as our analyses of data shaped our 
understanding of the impact of their activities in the schools. Thus, we have taken on the role of 
consultant, providing formative information as the Project has evolved. 

Inherent in our approach to evaluating the Learning Cormections Project are political, 
social, and philosophical stances. Those stances are not stated explicitly in our plan, but are 
implied in our methods, questions, and communications with the staff and with the teachers, 
principals, students, and parents involved. There are certainly some fundamental questions that 
go imasked in our approach in that we are not explicitly addressing the political, social, and 
historical inequities in the schools involved. What follows is a critique of mainstream evaluation 
efforts as they have transpired over recent years and an exploration of the consequences and 
benefits of explicitly addressing issues of power and equity in evaluation designs. 
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Serving a Particular Set of Social and Political Values 
Over the years, evaluation has come to be seen as a political undertaking. Michael Quinn 
Patton (1998) summarized 12 recent trends in evaluation; one of them was increasing political 
sophistication and acknowledgment of the role of values and morals in evaluation practice. There 
can be no doubt that evaluation is influenced partly by political forces and, in turn, that is has 
political effects. Whose interests are served and how interests are represented in any evaluation 
approach are now very critical concerns in a society with increasing disparate value systems, 
growing socioeconomic gaps, and multiplying linguistic and cultural diversity. 

In earlier days, evaluators assumed that the interests of all parties were properly reflected 
in the traditional outcome measures (postpositivist approach), but this assumption came to be 
questioned, and it was recognized that different groups might have different interests and might 
be differentially affected by the educational program and its evaluation (House, 1993). 
Stakeholders (those who had a stake in the program under review) became a common concept, 
and representing stakeholder views in the evaluation became an accepted practice. 

The stakeholder concept is based on the prevailing pluralist-elitist-equilibrium theory of 
democracy, which disclaims any normative judgements and which holds that the current system 
of competing parties and pressure groups performs the democratic function of equalizing the 
diverse and shifting political demands (MacPherson, 1987). It is perceived that describing what 
others value is the stance best suited to the political context in which evaluators operate, because 
decision making depends on the values held by relevant policy makers and stakeholders (House, 
1993). Presumably, these parties will use the findings to make informed decisions (pragmatist 
approach). Neither the government nor the evaluator is supposed to intervene to support any 
particular interests, but rather only to provide information that is value-neutral and interest- 
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neutral. The interests of various groups somehow dissolve into the values of decision makers and 
stakeholders. 

In two highly visible stakeholder evaluations funded by the federal government, those of 
Cities-in-Schools and Jesse Jackson’s PUSH/Excel program, the evaluations worked against the 
interests of the program participants and the inner-city students which the programs were 
supposed to serve, thus calling into question the justice of these evaluations (House, 1988; Stake, 
1986). The results of the PUSH/Excel evaluation were used not only to discredit the program but 
also to question Jesse Jackson’s ability to manage large enterprises during ensuing presidential 
campaigns. In truth, the stakeholder model was never implemented (House, 1988; Stake, 1986). 
Charles Murray, the evaluator in both cases, substituted a technocratic (postpositivist) approach 
to evaluation and expressed his disdain for the stakeholder concept in his article. Stakeholders as 
ncr.k Chairs (1983). Although the stakeholder approach seems firmly entrenched, there is 
disagreement about how to implement it. In reality, stakeholders do not have equal power to 
influence and utilize the evaluation, nor do they have equal protection from the evaluation 
(House, 1976, 1988, 1993). 

The problem of addressing multiple values and interests and how they should be 
represented in an equitable evaluation can take one directly into the realm of issues such as 
diversity, power, and equity. The consideration of these issues should be combined with the 
recognition of the assumptions, character, and consequences of conventional forms of 
educational evaluation. The problem of evaluation representing a particular set of social and 
political values (i.e., a broadly conservative set) also raises some serious questions about 
evaluation in general. Although the socio-political reality of multiple stakeholders and evaluators 
who have legitimate values and sometimes conflicting interests is recognized, how these values 
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and interests are legitimized will become one of the most important challenges for educational 
evaluation in the future (House, 1993). How to synthesize, resolve, and adjudicate all these 
multiple multiples in our increasing multicultural and amorphous society remains a formidable 
question, as indeed it does for the larger society. 

One thing we do know is that the socio-political reality in evaluation of public programs, 
both in education and health, often works in favor of higher income groups and against equity 
despite the stated goals and objectives of the social programs (Birdsall & Hecht, 1995; Paul, 

1991; Fine, 1983). When we look at the political structures and the broad organization of society, 
resource allocation and subsequent delivery of services and programs tend to be skewed in favor 
of those who have more voice (Fine, 1983; Fine & Weis, 1993). In many cases, powerful 
stakeholders or groups which are able to effectively demonstrate their interest in receiving social 
services and “effective” or “successful” social programs, manage to get the lion’s share of the 
resources and the funds. 

For purposes of elaboration, we will briefly touch upon the notion that the United States 
of America is one of the last Western industrialized nations to base their educational financing 
system on the taxation of largely differentiated property values (Berliner & Biddle, 1997). This 
financial arrangement alone should illuminate some of the deeper issues at stake in the evaluation 
of public education environments. American public school law and its case history has 
demonstrated time and again that there are very few instances where citizens have been able to 
prove that state school finance systems result in revenue disparities which violate the Equal 
Protection Clause of the Fourteenth Amendment (Alexander & Alexander, 1992). In 1973, in the 
case of San Antonio Independent School District v. Rodriguez, Mr. Justice Powell, delivered the 
opinion of the United States Supreme Court. He wrote, “to the extent that the Texas system of 
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school financing results in unequal expenditures between children who happen to reside in 
different districts, we carmot say that such disparities are the product of a system that is so 
irrational as to be invidiously discriminatory... ’’(Alexander & Alexander, 1992, p.779). 

If disparate allocation of governmental benefits can be justified on the basis of reasonable 
classification or the interests involved are not fimdamental, then statutes will be regarded as 
constitutional (Alexander & Alexander, 1992). The court in. the Rodriguez case basically ruled 
that a state legislature can heap benefits on some wealthy school districts and deprive others of 
fiscal resources and not offend the federal Equal Protection Clause. Thus, representing the 
educational interests of disenfi-anchised stakeholders, even within the American public school 
law domain, can be confounded with many inherently unequal and conflicting value systems. 

In other instances, our social service promoters, and social program managers in 
education and health, are able to shape the systems to serve their ovm personal and professional 
goals at the expense of equitable delivery (Paul, 1991). Problems created by the limited voice of 
politically weak or disenfi'anchised stakeholders are exacerbated in educational evaluation, when 
combined with direct provision of services in virtual public monopolies of the “best teachers”, 
the allocation of “best practices” in education, and the provision of high quality curriculum and 
professional development training which are centralized in higher socio-economic communities 
(Fine, 1983b, 1994; Oakes, 1986; Oakes & Guiton, 1995; Shapiro & Purpel, 1998). Ultimately, 
citizens have limited capacity to improve the public education they are provided through 
participating, informing, and making recommendations. This is especially true of lower socio- 
economic community stakeholders who have traditionally been limited in their capacity to have 
their voices heard without legal representation (Fine & Weis, 1993; Oakes & Guiton, 1995). 
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Historically, when interests have been ignored and educational procedures have been 
violated, lower socio-economic communities, minorities, exceptional populations, and limited 
English proficient citizenry have had to turn to the legal system for any kind of adjudication 
(Paul, 1991; McCormick, Haring, & Haring, 1990; Oakes & Guiton, 1995). Similarly, in terms of 
fighting for diversity, power, and equity in education, evaluation of education reform efforts 
could benefit from addressing some of the principals in American public school law (Alexander 
& Alexander, 1992). Regardless, it is clear that contemporary evaluation approaches, which were 
invented to solve social problems, can be afflicted with many of the problems they were meant to 
solve. 

One must also note carefully that today’s professional evaluators sometimes become 
evaluators by default. We represent an eclectic and diverse combination of various professional, 
academic, and research areas. Shadish and Epstein (1987) found that 31% of the respondents in 
their survey described their primary professional identity as that of “evaluator” (p. 560). Others 
thought of themselves first as a psychologist, sociologist, economist, educator, and so on, with 
identity of evaluator secondary. When both Charles Murray (1983, 1984) and Michele Fine 
(1983b, 1988) have been successful evaluators representing particular, divergent sets of social 
and political values and interests, one has to acknowledge the diverse socio-political reality in 
which evaluators actually find themselves in practice. 

The socio-political reality in which one can survive as an professional evaluator of 
education reform becomes integrated into a world with those individuals that agree with your 
views on the nature of evaluation research and its influence on decision making processes. As 
evaluators conduct their research to address their prospective typical questions and key audiences 
with their preferred methods, they will demonstrate either explicitly or implicitly their particular 
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set of ideological, philosophical, and value orientations. Surely, society expects evaluation to be 
based on scientific authority. However, the notion of what is scientific has been the subject of 
much debate (e.g., Harding, 1991; Toulmin, 1960). The concepts of objectivity, scientific 
methodology, and validity will be recast to continue to accommodate different philosophical and 
political stances and varying evaluation approaches (House, 1993). 

Evaluation research of education reform efforts can contribute to informed decision 
making, but the manner in which this is done needs to be reformulated. We are well past the time 
when it is possible to argue that good research will, because it is good and rigorous, influence the 
decision making process. Rist (1998), the Evaluation Advisor to the Economic Development 
Institute of the World Bank, argues succinctly that this kind of linear relation of research to 
action simply is not a viable way in which to think about how knowledge can inform program 
managers, reformers, or policy makers. The relation is both more subtle and more tenuous. 

Consequences and Benefits of a Critical Normative Evaluation Approach 

Schools are inextricably linked to the communities they serve through social, political, 
economic, and cultural interests. To better comprehend public education, the socio-cultural, 
political, and hierarchical relationships that transpire within the school as well as within the 
community must be linked to the broader political and economic issues of society at large (Ogbu, 
& Matute-Bianchi, in press). To begin to realize the possibilities for reforming public education, 
and to begin fighting for diversity, power, and equity in education, especially for those children 
who are disadvantaged, we must first re-examine the historical nature of the problems of 
education and the communities in which these schools exist (Noll, 1997). 

Education Reform 
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School reform remains at the center of the public agenda even after many years of 
discussion, legislation, and state and local action. After years of work to improve public 
education, student achievement for many populations still remains below acceptable levels. This 
is p 2 irticularly true for populations who traditionally have been poorly served by our schools. For 
example, on the 1994 National Assessment of Educational Progress reading assessment, 20% of 
white fourth graders scored below the basic level in reading, but 69% of African American 
students and 64% of Hispanic students scored poorly (NWREL, 1998). 

The statement “All men are created equal” is one that resounds throughout American 
history. The words are found in the Declaration of Independence and Lincoln’s Gettysburg 
Address; they are also paraphrased and applied in numerous settings. For educators and 
educational evaluators, it has meant that American schools are charged with offering every child 
equality of educational opportunity. This concept of equality is one that has been implicit in most 
educational practices throughout the period of public education in the nineteenth and twentieth 
centuries (Coleman, 1981). However, few Whites in suburban America would long tolerate the 
low academic achievement taken for granted in the urban or rural public schools attended largely 
by African Americans, Hispanics, and impoverished children. 

In big cities all over the U.S., minority students by the tens of thousands leave school 
each year, some as dropouts, some as graduates, utterly unprepared to participate in and 
contribute to a democratic society (Oakes & Sirotnik, 1986). They lack the skills that will allow 
them to obtain gainful employment, and they are devoid of the preparation that will lead to 
success in further education. Freire (1985) would characterize this lack of skills and preparation 
as the “inability to act upon and transform one’s world” (cited in Hamnett, Kumar, Porter & 
Singh, 1984, p. 68). Consequently he would say that the democratic society failed to move this 
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person toward the ever-new possibilities of a fuller and richer life individually and collectively 
through the hospices of public education (Hamnett et al., 1984). 

Reform reports in the 1980s and early 1990s served to spotlight the nature and the 
function of public schooling and attempted to delineate a specific relationship between broader 
social, economic, political, and cultural interests. Kretovics, Farber, and Armaline (1991) state 
that many of the reports argued in favor of a wide range of additive reforms such as increased 
testing, more homework, a longer school year, a longer school day, and the internalization of an 
extensive list of cultural facts. Others suggested a plethora of technical solutions for the 
challenges facing public schools, ranging from the addition or reduction of certain educational 
requirements for teachers to the addition of specific course requirements to the public school 
curriculum (Noll, 1997). For the most part, the reform efforts have been driven by a persistence 
of list logic and an obsession with the quick fix (Barth, 1986; Giroux, 1983). 

Berliner and Biddle (1997) suggest that there are data that illuminate the untenable 
assumptions on which much of the school reform movement is based. Similar to Kozol (1991), 
they also suggest that some of these reform efforts are thinly disguised elitist attempts to get rid 
of public education, to protect the privilege such individuals have already bestowed upon their 
children. Kozol (1991) also suggests that much of the school reform movement is wrongheaded 
or ineffective. Coming from a radical social theorist perspective, these claims might be 
completely dismissed, but Berliner is identified by the scientific community as a respected, 
traditional educational psychologist. Some proposals for education reform reflect only the 
personal experiences or prejudices of legislators, sponsors, and local humanitarians, and some are 
based on misunderstandings about schools and the problems of education (Barth, 1986; Oakes, 
1986; Shapiro & Purpel, 1998). For these reasons, many programs intended to improve our 
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schools turn out to have little detectable effect or, worse, end up creating serious problems for 

•V, 

educators and students. 

For more than 20 years, a variety of educational and social theorists have presented 
compelling arguments that illustrated the reproduction of social, economic, political, and cultural 
inequalities through the organization and structure of the schooling process (Kretovics & Nussel, 
1994). They summarize some of the past characters that entered the debate on controversial 
issues in education and social theory. Kretovics and Nussel (1994) state that educators from 
diverse cultural and ideological backgroimds (i.e., Coleman, Anyon, Giroux, Amot, Clark, 
McRobbie, Illich, Bowles, Gintis, and Apple), drawing on the earlier works of other scholars 
(i.e., Dewey & Baldwin), have pointed to the political and ideological nature of schooling and the 
ways in which schools often imder serve the nonmajority students and, through hegemonic 
practices, reproduce the status quo. These authors say that the arguments have often been 
ignored, dismissed, or co-opted, resulting in a blaming of the victims of educational inequalities. 
As a result, schools in general are blamed for the broader social and economic problems that 
inform and structure their existence. 

Through the recent school reform movement, the problems of American education and 
the general purposes of public schooling have been systematically removed from the terrain of 
public debate (Kretovics & Nussel, 1 994). Kretovics and Nussel go on to say that “most of the 
widely publicized school reform efforts have created an educational climate antithetical to moral 
referents such as social justice and initiatives of equity that are valued in a democratic society” 
(1994, p.4). Giroux (1983) and Freire (1985) believe that the schooling process is always 
structured on the norms and values that embody specific social, political, economic, cultural, and 
ideologic interests. Therefore, as evaluators of educational programs, we should critically 
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examine the context, both present and past, in which education takes place in order to illuminate 
both the problems and the possibilities for change in the future. Evaluators of education reform 
must have a vision for change and the ability and conviction to act on that vision. Evaluators are 
inevitably linked to one of the most crucial of social processes, education, and must develop a 
framework that takes seriously issues of diversity, power, equity, as well as educational 
structures and practice, in the difficult process of reforming.American public education (Shapiro 
&Purpel, 1998). 

Facing our Ideologies and Reflecting on our Situational Ethics 

Denzin and Lincoln (1998) state that explicit recognition of the ideological contours of 
program evaluation did not always exist. Rather, both inside and outside the field, a methods 
orientation has predominated. One continuing legacy of this orientation is the naming of the 
different evaluation approaches by their primary methods and their key audiences. Yet what 
importantly distinguishes one evaluation methodology from another is not the methods, but 
rather whose questions are addressed and which values are promoted (Denzin & Lincoln, 1998). 
Program evaluators continue to try and distance themselves from the political dimensions of their 
work, intentionally seeking the objective stance of politically neutral and scientific rationality. 

Weiss (1987) has most influentially critiqued this distanced stance, arguing that it 
substantially underestimates the potency of evaluation in this era. She has maintained, for 
approximately three decades, that social policy and program decision making are not rational 
processes to which data-based enterprises such as program evaluation could contribute the 
definitive piece of information. Rather, “politics of program survival and the politics of higher 
policymaking accord evaluative evidence relatively minor weight in the decisional calculus” 
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(Weiss, 1987, p.62). In short, neither distanced objectivity nor neutral rationality is going to earn 

program evaluators a seat at the decision making table. 

American evaluators are renovmed globally for their trend-setting potential in evaluation 

theory, practice, methods, and policies. Jan Eric Furubo, Secretary General of the European 

Evaluation Society, characterizes our evaluation development in the following terms: 

Evaluation is not a rigorous clinical science; rather it is part of the art and craft of public decision making. 
To be doing evaluations is seen as a badge of modernity and rationality, much of its value is symbolic. So 
if evaluation is to put down roots it has to demonstrate that it can serve other purposes as well as the 
symbolic. That other value surely has to be that of assisting decision makers, often assisting them in 
making painful decisions in difficult circumstances (http://www.europeanevaluation.org/no2_97.htm). 

In sharing this quote from the Secretary General, we hope to remind American evaluators of their 
own genesis and context in a global society. This reflection and reminder can help reveal our 
situational and generalized ethics that are bound by the relationships and ideological orientations 
we share with our clientele. Maintaining distanced objectivity and neutral rationality under the 
guise of a rigorous and good science for program evaluation fails because it focuses on the truth 
aspect of validity to the exclusion of the credibility and normative aspects of social programs. By 
the essence of his or her social function, “the evaluator is engaged with the world and can 
directly affect who gets what” (House, 1980, p. 254). Further, as an inherently political activity, 
evaluation “is intimately implicated in the distribution of basic goods in society. ..so evaluation 
should not only be true; it should also be just” (p.l21). 

Using Critical Normative Evaluation in Education Reform 

Understanding what role our own ideologies and situational ethics play in program 
evaluation of education reform is only the first step to learning how to reflect and communicate 
issues of power in evaluation. The second step is in knowing that there is a space for many kinds 
of science, logical analysis, and truth seeking research, that one kind of evaluation research that 
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can be just as rigorous as the next. The critical normative science approach to evaluation 
provides a framework for making explicit the political and philosophical meanings of the 
evaluation questions, and the meanings of our ideological and value orientations. These 
meanings can and should be conununicated and negotiated to high level and mid-level policy 
makers, to program managers, program beneficiaries, their communities, and other less powerful 
groups before an evaluation relationship with potential clients is formalized. 

For purposes of illustration only, we will explore how the nature and function of the 
evaluation for Learning Connections would change under the auspices of a critical normative 
evaluation framework. In order to place the evaluation study within this framework, one must 
first focus on the issue of evaluation questions. As argued before, whose questions are addressed 
as the evaluation plan is formulated and which particular values and interests are promoted are 
what differentiates one evaluation methodology from another. Thus, in order to reflect the typical 
evaluation questions of a critical normative approach, one of the questions about the project 
could be, “In what ways are the assumptions and/or activities of the program serving to maintain 
the inequities in educational opportunity?” and “In what ways are the premises and goals of 
Learning Connections serving to maintain the power and resource inequities in the conununity?” 
The implications for answering these questions are especially relevant to the purpose of public 
schooling; to provide equal and excellent education for all children in a democratic society. 

These types of questions are very familiar to the feminist, neo-Marxist, critical social theories 
that form both the philosophical and epistemological foundations for the critical normative 
approach to social program evaluation in the public domain. 

In addition to changing the questions, other components of the research design would be 
augmented. A representative sample of program beneficiaries such as the African American and 
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impoverished students, their parents, and other community members would be a priority to 
include in the research sample and in the research design, by actively recruiting them for 
activities such as formulating goals and objectives, as well as participating in varied structured 
and unstructured interviews. Also, an in-depth analysis of student demographic data from schools 
would be conducted to determine the nature and extent of the classroom and campus pedagogical 
challenges to instructional organization. For instance, teachers face many challenges in their daily 
practice with students in regular, special, gifted, and compensatory education (e.g.. Chapter I, 
Title I, and 504). Investigating the individual and group variation in these educational services 
would prove beneficial in developing a classroom and campus image of how teachers experience 
exceptional and at risk students before, during, and after the social program implementation. In 
addition, some case studies involving a thorough document (content) analyses of the enacted 
curriculum in particular classrooms across a school day, week, and 9-week grading period could 
prove informative for more long-term determinations of instructional change and improvement. 
These types of data collection would be pursue in order to establish baseline information about 
instructional and curriculum practices within classrooms and among schools. 

The above evaluation would seek to illuminate the historical, structural, and value bases 
of the social phenomenon in local schooling practices and, in doing so, would aim to foster 
greater justice, diversity, equity, and power in educational opportunity. In such a case, the 
Learning Connections Project, its premises, goals and objectives, and its implementation would 
be evaluated using the lens of a critical normative eye. Such an approach to evaluation would be 
designed to provide information to disadvantaged students, many of whom possess nontraditional 
learning profiles (Gross, 1993), to their parents, and to other stakeholders (e.g., teachers, 
principals) in the interest of their taking action to ensure their students’ opportunity to learn and. 




28 



Voices in Evaluation 28 



eventually, to participate fully in a democratic society. Hence, in the endeavor of opening spaces 
for a different kind of logical analysis, this approach would capitalize on an “openly ideological” 
form of inquiry (Greene, 1998, p. 377). In this form of inquiry and investigation, the political 
aspects of the voices in the evaluation would be explicitly brought into the discourse of the 
Learning Connections Project. 

Clearly, the communication to possible reformers of the explicit questions, orientations, 
and paradigms that this approach emphasizes is essential for negotiating the social and political 
values of emancipation, empowerment, and social change within the instructional context of 
public education. National policymakers, educational leaders, education reformers, and children 
in disadvantaged situations can benefit from critical normative evaluation, but not in the same 
ways and not with the same evaluator roles as in other evaluation frameworks. The consideration 
of an evaluation’s questions and the evaluator’s own ideology and value orientations all need to 
be made explicit before a formal evaluator-client relationship is in place. The challenge will be to 
create appreciation and space for such diversity among both those within and outside the 
profession who have a single and narrow view of evaluation and its practice (Worthen, Sanders 
& Fitzpatrick, 1997). The debate will and should go on so that we can discover implications and 
ramifications of diverse approaches. We foresee no desire to turn back the clock to a single 
dominant perspective. 

Conclusion 

Irrespective of the many social, economic, technological, cultural, and political problems 
that face our American communities, the public schools exist for the purpose of educating all 
children. Teachers are a part of the never-ending struggle to create conditions in which 
meaningful learning takes place and to provide the best educational opportunities in a given 




29 



Voices in Evaluation 29 



situation. As evaluators rendering judgement on educational programs, and giving merit or not 
giving merit to the educational repertoires and learning outcomes of teachers, we also become 
inextricably linked to the process of either perpetuating a disconnected reality of education or 
critically examining and observing a wide range of crucial issues, structures, and problems in 
contemporary education. As evaluators of education programs and teaching, we cannot ignore 
that we become a part of the never-ending struggle to make judgments about a social activity 
which creates the conditions for or obstacles to social mobility. 

The central task of the current reform movement in education is nothing less than 
building and transforming schools that are struggling to achieve democratic ideals (Fine, 1994). 
While schools can be described as potentially a site of extraordinary democracy, the processes 
and outcomes of schools deeply reproduce and promote the very social inequities they are said to 
equalize (Fine, 1983). This circumstance imposes onto the roles of educational leaders and 
evaluators a social responsibility, one that demands sincere conscience and deliberate action. 
Evaluators and researchers, who in the past have been content to describe dispassionately what 
schools are doing and how they are functioning, are actually involved in and should be 
committed to a collaborative view of knowledge creation. When evaluators of education reform 
gain the conscience and purposefulness of their critical role, no relationship is left untouched or 
unchanged. 

In conclusion, evaluation is a powerful social force that has evolved only recently in advanced capitalistic 
societies, a new institution that promises to be a major influence over the long term. Its influence can be 
both good and bad. In either case, society before formal evaluation is not the same as society afterward. 
Exactly what shape the practice, institution, profession, and discipline will take in the future is impossible 
to predict. What is clear is that its fate will be bound to the government and the economic structure and 
determined in part by its own history and traditions. Some of its destiny lies within the control of the 
evaluators themselves; some does not (House, 1993, p.l72). 
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